COMP: use sha512 instead of md5 hashes for ExternalData fetching #1237

bcdarwin · 2021-09-15T21:03:48Z

Description

Not finished yet since I don't have all the files on my system in order to compute their hashes.

Fixes #1236 when complete.

…e API change

cookpa · 2021-09-15T21:18:55Z

This is great @bcdarwin, thanks

bcdarwin · 2021-09-16T16:27:14Z

To clarify, I can't convert the remaining .md5 files since I don't have copies of the remaining files needed (which I can no longer download), so someone with a recent build needs to finish this or send me the build tree.

cookpa · 2021-09-16T16:37:50Z

Do you have a list of files you need? I'm not sure I have them but can take a look

bcdarwin · 2021-09-16T16:41:21Z

Do you have a list of files you need? I'm not sure I have them but can take a look

The data files (or just their sha512sums) corresponding to the following checksums:

TestData/Data/phantomDwmgm.jpg.md5
TestData/Data/B5.tiff.md5
TestData/Data/mov2.nii.gz.md5
TestData/Data/functional.nii.md5
TestData/Data/r16priors.nii.gz.md5
TestData/Data/README.md5
TestData/Data/phantomFwmgm.jpg.md5
TestData/Data/myview2.csv.md5
TestData/Data/B2.tiff.md5
TestData/Data/ref2.nii.gz.md5
TestData/Data/phantomtemplate.jpg.md5
TestData/Data/listPriorWolk_nirepDemo.txt.md5
TestData/Data/r27slice.nii.gz.md5
TestData/Data/wolk_mask.nii.gz.md5
TestData/Data/myview3.csv.md5
TestData/Data/wolk_dataMCINormalDemo.raw.md5
TestData/Data/B4.tiff.md5
TestData/Data/priorScaleListWolk_nirep0p1Demo.csv.md5
TestData/Data/phantomBwmgm.jpg.md5
TestData/Data/B3.tiff.md5
TestData/Data/ford.jpg.md5
TestData/Data/myview1.csv.md5
TestData/Data/test_image_ground_truth.nii.gz.md5
TestData/Data/phantomAwmgm.jpg.md5
TestData/Data/ch2brainmask.nii.gz.md5
TestData/Data/test_image_seg.nii.gz.md5
TestData/Data/phantomCwmgm.jpg.md5
TestData/Data/wolk_dataMCINormalDemo.mhd.md5
TestData/Data/test_DiReCT.sh.md5
TestData/Data/B1.tiff.md5
TestData/Data/phantomEwmgm.jpg.md5
TestData/Data/myview4.csv.md5
TestData/Data/r85slice.nii.gz.md5
TestData/Data/r16mask.nii.gz.md5
TestData/Data/r30slice.nii.gz.md5
TestData/Data/myview_mismatch.csv.md5
TestData/Data/r62slice.nii.gz.md5
TestData/Data/phantomHwmgm.jpg.md5
TestData/Data/phantomGwmgm.jpg.md5
TestData/Data/beetlerot.jpg.md5

bcdarwin · 2021-09-16T16:46:35Z

Most likely not all of these are necessary for the default build, but at least some of them are.

cookpa · 2021-09-16T17:33:59Z

@ntustison @hjmjohnson is there an easy way to get at these files? I can't find a local copy

hjmjohnson · 2021-09-16T17:36:48Z

@ntustison @hjmjohnson is there an easy way to get at these files? I can't find a local copy

They should be available on midas. Which MD5 files do you need? I probably have them cached on my local computer somewhere.

hjmjohnson · 2021-09-16T17:39:09Z

@bcdarwin Is there a script used to do this conversion?

bcdarwin · 2021-09-16T20:05:25Z

Here's a Python script that can walk the ANTs source tree creating new checksum files:

#!/usr/bin/env python3


import glob
import os
from   os import path
import subprocess


filenames = {}

# accumulate all .md5 files
for f in glob.glob("TestData/**/*.md5", recursive=True):
  base, _ = path.splitext(path.basename(f))
  filenames[base] = f

# search for files with filenames matching the filename of a .md5 file and create a .sha512 file
for d, _, fs in os.walk('.'):
  for f in fs:
    name = path.basename(f)
    if name in filenames:
      out = filenames[name][:-4] + ".sha512"
      subprocess.check_call(f'sha512sum {path.join(d, f)} > {out}', shell=True)

cookpa · 2021-10-06T14:44:57Z

I found a few of the missing files

d5bcdd4c68e840c7d7bcac32e68ae7a20d2896ecb78f7d3dc2512707d88092b154a81aeb62392460fbd487f60248680a5c4777e7efe430a8b6fdcdd823f222af  Frown.nii
20942c444831f9dd71d41809acf0eac6fb351bda9a6e2d6826a645121b09d781a7feacdcd5edf4b7e9708f9e5de380bdfdfcb9f7c4b25fa1a03d506d817c3e1d  Frown.txt
6a5395aa5619d69802304998dde7d9b44d4e768725cb0bfd3cca20666a22d51c9879558d1767564f66de82c87c3b80df5e04338e84ee1299ff2876254fd1904f  Frown.vtk
a764b6b2e4d1b07c6ad0794973a45ed4967c86ce6dd84471000712be0bd9a5a0a405740e4af28b9909060b00ebe66a343e34d38affa719b54ac7c0180e9babd3  Smile.nii
0f6c76fb3222490b283500bb19d9fddb63a5670af660c25a6ecb181d2bde7036c5781898ad64b93f4918004adbdf62063e4bac79462cdb708acf34331c205e03  Smile.txt
4ca0ce32b28329ab42c0f7ef1816dc8e54978ac60573b14b75a53a93a6af42995aadd4451ddfd19fa0ab707feff1388a86fa675c558ace1906a1ff867b15281f  Smile.vtk

bcdarwin · 2021-10-06T17:49:56Z

@cookpa - thanks; I've updated the PR and the list of missing files above.

cookpa · 2021-10-06T19:06:16Z

Thank you @bcdarwin . I found some more in the Kitware ARCHIVEOBJECTS collection

97c0f6cf99602d29a0c5070f61b8fa3348f492b6f04bf4b60eea34f369e6b19b8f9405c783df27d4e5bc1d0071081a4ea93f6725caa61bddaee1e1e9723b13d8  CompositeTransformUtilTest.result.h5
33b868a72ef3b3ae416582b0572bf519e19e2a1afd8c0fc475db06fe2c34a83d35b9008d938e4141cebc60b781442fc71de1563b6e11f8c93c5d5db8fb3e612d  CompositeTransformUtilTest_AffineTransform.mat
e847817e32a08f5efbdf038a2cba7c17c39906d14fc2ced53736f47dda8c2f94df8c9611875924bd4e6055678ff490f44719ae013d76626f1bbe5e208eb48123  CompositeTransformUtilTest_RigidTransform.mat
969736df0271840d8f1dac90ce871b670de95215a673d5e64cf4c37eeffd71fd631059a5354ef263afb94335b989fa94aa2701c783509d392d0cc262ea1adfcc  CompositeTransformUtilTest_SyNTransform.nii.gz
ded1346ac2813dc2afabf96654812a773519a3c0b2d700213d0997dee74d87c8cc734e176f4444bf9b1ec64b862a8007fcdf06114a056fbc7093dd8fd634438f  Initializer_antsRegistrationTest_TranslationRescaleHeadMasks_Float.mat
2e2de8f219093c3c7f5c9e175820bc633925dc22a49a8707370d47b45ecc64c7cadb98aa4cd012878e9b42dc28f13feb9aa69b151b0710d635c4091d026ee251  antsRegistrationTest_SimilarityRotationNoMasks.result.nii.gz
ce8f2e5fdaba35a6e25de1969f7bd34a34dd9201c9e05181bf69d557a2c187a768e14cbc513400e45e72f71c5464c81753e33476f1e3e1838e4663f6cef9d75e  antsRegistrationTest_SimilarityScaleMasks.result.nii.gz
14e8a14b7177e090cf4286337aabd757e37f3e6d7f634e3697bb8f77dc85ad44f72619ec38b691c04e301cf262d3933a42b4aad01ae3da0adb31ddd4ce07c0f7  antsRegistrationTest_SimilarityScaleNoMasks.result.nii.gz
ea791e9f3f6173b6ecef0aff56cbab594b0363658c31829c2822a8af252a269486e64122af9122916094bee5ffac78729d6737e7b30602e3e91478b35abb0d02  antsRegistrationTest_SimilarityTranslationRescaleNoMasks.result.nii.gz
6aa6b5366c359679cadf7ced45a2e76b38533f0eb89a668a2b52bf7a37e0910a6b6172dfd243937e192def14256f391405b72d262ba603f9ad67e9cb446cc800  antsRegistrationTest_SimilarityTranslationRescaleNoMasks_Float.result.nii.gz

…Kitware API change

bcdarwin · 2021-10-06T22:29:33Z

@cookpa @hjmjohnson after incorporating the latest batch of file hashes the default build succeeds. Therefore I suggest you consider merging this PR as-is and fixing the remaining downloads listed above later.

cookpa · 2021-10-06T23:40:08Z

I think that's a good plan. I guess the other files were hosted on one of the test servers that were deprecated already? I can't find them at data.kitware.com or at www.itk.org. I don't know how to access files at slicer.kitware.com.

jzhang-neuro · 2022-05-02T17:04:44Z

@cookpa @hjmjohnson after incorporating the latest batch of file hashes the default build succeeds. Therefore I suggest you consider merging this PR as-is and fixing the remaining downloads listed above later.

Hello, experts! @cookpa @bcdarwin I have tried to change the hash md5 to sha512 as shown in 221dad7, but still failed in two files (the others seem worked well): Object SHA512=875b7adb24f8958648a4b6ec495dcf00, and Object SHA512=42cd2424c063ac3be1aa469ad4f1d2db. They are just the two files that you marked as File renamed without changes: Initializer_0.05_antsRegistrationTest_SimilarityRotationNoMasks and Initializer_0.05_antsRegistrationTest_SimilarityScaleNoMasks. Should they change the content when converting from md5 to sha512? If so, what's the sha512 hash value for the two?

Here is the complete error code:
`CMake Error at /usr/local/share/cmake-3.22/Modules/ExternalData.cmake:1181 (message):

Object SHA512=42cd2424c063ac3be1aa469ad4f1d2db not found at:

file:////home/mr_zyq/Applications/ANTs-2.1.0/ANTs/.ExternalData/SHA512/42cd2424c063ac3be1aa469ad4f1d2db ("Couldn't read a file:// file")
http://slicer.kitware.com/midas3/api/rest?method=midas.bitstream.download&checksum=42cd2424c063ac3be1aa469ad4f1d2db ("HTTP response code said error")
https://data.kitware.com:443/api/v1/file/hashsum/SHA512/42cd2424c063ac3be1aa469ad4f1d2db/download ("HTTP response code said error")
http://www.itk.org/files/ExternalData/SHA512/42cd2424c063ac3be1aa469ad4f1d2db ("HTTP response code said error")

Examples/TestSuite/CMakeFiles/ANTSFetchData.dir/build.make:328: recipe for target 'ExternalData/TestData/Initializer_0.05_antsRegistrationTest_SimilarityScaleNoMasks.mat-hash-stamp' failed
make[5]: *** [ExternalData/TestData/Initializer_0.05_antsRegistrationTest_SimilarityScaleNoMasks.mat-hash-stamp] Error 1
make[5]: *** Waiting for unfinished jobs....
CMake Error at /usr/local/share/cmake-3.22/Modules/ExternalData.cmake:1181 (message):

Object SHA512=875b7adb24f8958648a4b6ec495dcf00 not found at:

file:////home/mr_zyq/Applications/ANTs-2.1.0/ANTs/.ExternalData/SHA512/875b7adb24f8958648a4b6ec495dcf00 ("Couldn't read a file:// file")
http://slicer.kitware.com/midas3/api/rest?method=midas.bitstream.download&checksum=875b7adb24f8958648a4b6ec495dcf00 ("HTTP response code said error")
https://data.kitware.com:443/api/v1/file/hashsum/SHA512/875b7adb24f8958648a4b6ec495dcf00/download ("HTTP response code said error")
http://www.itk.org/files/ExternalData/SHA512/875b7adb24f8958648a4b6ec495dcf00 ("HTTP response code said error")

Examples/TestSuite/CMakeFiles/ANTSFetchData.dir/build.make:321: recipe for target 'ExternalData/TestData/Initializer_0.05_antsRegistrationTest_SimilarityRotationNoMasks.mat-hash-stamp' failed
make[5]: *** [ExternalData/TestData/Initializer_0.05_antsRegistrationTest_SimilarityRotationNoMasks.mat-hash-stamp] Error 1
CMakeFiles/Makefile2:6095: recipe for target 'Examples/TestSuite/CMakeFiles/ANTSFetchData.dir/all' failed
make[4]: *** [Examples/TestSuite/CMakeFiles/ANTSFetchData.dir/all] Error 2
Makefile:165: recipe for target 'all' failed
make[3]: *** [all] Error 2
CMakeFiles/ANTS.dir/build.make:87: recipe for target 'ANTS-prefix/src/ANTS-stamp/ANTS-build' failed
make[2]: *** [ANTS-prefix/src/ANTS-stamp/ANTS-build] Error 2
CMakeFiles/Makefile2:866: recipe for target 'CMakeFiles/ANTS.dir/all' failed
make[1]: *** [CMakeFiles/ANTS.dir/all] Error 2
Makefile:120: recipe for target 'all' failed
make: *** [all] Error 2`

cookpa · 2022-05-02T17:18:52Z

@JayChang91 please open an issue using the issue template for build problems, and provide the full information about what you ran to get the above error. You can link to this PR by mentioning its number "#1237".

bcdarwin marked this pull request as draft September 15, 2021 21:03

COMP: use sha512 instead of md5 hashes for ExternalData due to Kitwar…

b77b499

…e API change

bcdarwin force-pushed the use-sha512-hashes branch from 7f1ce05 to a41d9b8 Compare September 15, 2021 21:06

cookpa mentioned this pull request Oct 6, 2021

Error building ANTs related to MD5 #1248

Closed

bcdarwin force-pushed the use-sha512-hashes branch from a41d9b8 to a19d604 Compare October 6, 2021 17:47

COMP: start converting ExternalData hashes from md5 to sha512 due to …

57ea2b5

…Kitware API change

bcdarwin force-pushed the use-sha512-hashes branch from a19d604 to 57ea2b5 Compare October 6, 2021 20:38

bcdarwin changed the title ~~[WIP] COMP: use sha512 instead of md5 hashes for ExternalData fetching~~ COMP: use sha512 instead of md5 hashes for ExternalData fetching Oct 6, 2021

bcdarwin marked this pull request as ready for review October 6, 2021 22:27

cookpa merged commit 221dad7 into ANTsX:master Oct 7, 2021

bcdarwin deleted the use-sha512-hashes branch October 7, 2021 15:31

jzhang-neuro mentioned this pull request May 9, 2022

Build error: Object SHA512 for Initializer_0.05_antsRegistrationTest_SimilarityRotationNoMasks and Initializer_0.05_antsRegistrationTest_SimilarityScaleNoMasks #1356

Closed

cookpa mentioned this pull request Mar 28, 2023

Download testing data manually #878

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

COMP: use sha512 instead of md5 hashes for ExternalData fetching #1237

COMP: use sha512 instead of md5 hashes for ExternalData fetching #1237

bcdarwin commented Sep 15, 2021

cookpa commented Sep 15, 2021

bcdarwin commented Sep 16, 2021 •

edited

Loading

cookpa commented Sep 16, 2021

bcdarwin commented Sep 16, 2021 •

edited

Loading

bcdarwin commented Sep 16, 2021

cookpa commented Sep 16, 2021

hjmjohnson commented Sep 16, 2021

hjmjohnson commented Sep 16, 2021

bcdarwin commented Sep 16, 2021 •

edited

Loading

cookpa commented Oct 6, 2021

bcdarwin commented Oct 6, 2021

cookpa commented Oct 6, 2021

bcdarwin commented Oct 6, 2021

cookpa commented Oct 6, 2021

jzhang-neuro commented May 2, 2022 •

edited

Loading

cookpa commented May 2, 2022

COMP: use sha512 instead of md5 hashes for ExternalData fetching #1237

COMP: use sha512 instead of md5 hashes for ExternalData fetching #1237

Conversation

bcdarwin commented Sep 15, 2021

Description

cookpa commented Sep 15, 2021

bcdarwin commented Sep 16, 2021 • edited Loading

cookpa commented Sep 16, 2021

bcdarwin commented Sep 16, 2021 • edited Loading

bcdarwin commented Sep 16, 2021

cookpa commented Sep 16, 2021

hjmjohnson commented Sep 16, 2021

hjmjohnson commented Sep 16, 2021

bcdarwin commented Sep 16, 2021 • edited Loading

cookpa commented Oct 6, 2021

bcdarwin commented Oct 6, 2021

cookpa commented Oct 6, 2021

bcdarwin commented Oct 6, 2021

cookpa commented Oct 6, 2021

jzhang-neuro commented May 2, 2022 • edited Loading

cookpa commented May 2, 2022

bcdarwin commented Sep 16, 2021 •

edited

Loading

bcdarwin commented Sep 16, 2021 •

edited

Loading

bcdarwin commented Sep 16, 2021 •

edited

Loading

jzhang-neuro commented May 2, 2022 •

edited

Loading